Skip to content

CI: Add Alpine Linux 3.23 runner to the pipeline#18087

Merged
tonyhutter merged 1 commit intoopenzfs:masterfrom
alex-moch:alpine-ci
Dec 30, 2025
Merged

CI: Add Alpine Linux 3.23 runner to the pipeline#18087
tonyhutter merged 1 commit intoopenzfs:masterfrom
alex-moch:alpine-ci

Conversation

@alex-moch
Copy link
Contributor

Add an Alpine Linux 3.23 runner to the CI chain to run OpenZFS builds and tests against musl libc.

Currently, zfs_send_sparse is killed after ~10 minutes on Alpine, causing cascading EBUSY failures in the test suite. With zfs_send_sparse disabled, the ZFS test suite reaches a pass rate of 94.62%.

This commit introduces the required Alpine-specific setup and a small set of shell and cloud-init compatibility fixes that also apply to existing Linux runners.

The Alpine runner is not enabled by default and is not executed for new pull requests.


Motivation and context

The CI chain currently does not cover musl-based Linux systems. Bugs on these systems are therefore often reported after new releases. This pull request is meant to catch such issues earlier, before they hit downstream.

Disclaimer: I am co-maintaining the sys-fs/zfs package for Gentoo and, even though musl-based profiles are marked as experimental, I care about this use case.

Incidentally, work on this pull request already resulted in the following bug fix:

So, I assume the Alpine CI does what it is supposed to do. 😉

Description

Parts of this commit are opinionated. I try to pick the simplest solution, keep the diff small, and avoid duplicating code where possible. If a maintainer prefers a different approach, I am very open to adjusting this.

The runner passes 94.62% of the ZFS test suite with zfs_send_sparse disabled. This is a solid starting point, but not sufficient for a green CI run. zfs_send_sparse currently times out and causes EBUSY failures that cascade through the test suite. There are also unresolved issues around atime and hostid.

While a fully green CI would be desirable prior to enabling the runner, this pull request deliberately limits its scope to adding Alpine support without enabling it for new pull requests. As such, the primary risk of merging lies in a small set of compatibility changes that affect existing runners.

94.62% is a good baseline to continue improving the runner and eventually enable it once it reliably passes the full ZFS test suite.

What has been changed?

It is important to distinguish between changes that affect existing runners and those that are specific to the new Alpine runner. This commit does not touch zfs-qemu.yml. Therefore, the Alpine runner will not be executed when merging this pull request.

Changes affecting current runners

Some compatibility changes required for Alpine also affect the other runners in the CI chain. Most notably, the cloud-init configuration was adjusted. Aside from minor formatting changes, the relevant points are:

  1. Replace $BASH with /bin/bash.
    • $BASH is not set on Alpine. Bash is located at /bin/bash on all Linux distributions in the CI chain.
  2. Allow root to execute sudo without a password.
    • This appears to be the default on most distributions, but not on Alpine.
  3. Add lock_passwd: false and passwd: '*'.
    • This is a pragmatic fix. The Alpine guide suggests installing openssh-server-pam and enabling PAM in sshd_config. Given that this runs in ephemeral CI VMs, this approach keeps the configuration simple and avoids per-distro branching.
  4. Add bash and sudo packages.
    • These are not installed by default on Alpine, but are present on the other distributions.

The fourth point relies on apt for Debian-based systems, so cloud-init status --wait is needed to avoid hanging due to an apt lock. Adding this globally caused AlmaLinux 8 to hang, so it is limited to Debian and Ubuntu. Overall, waiting for cloud-init improves robustness.

In some scripts the Alpine runner initially complained about bash not being found, likely due to environment differences. These calls were changed to POSIX-compatible sh, as they are only used for output redirection with sudo.

Alpine mounts loop devices as /dev/loop/X instead of /dev/loopX, which required a small regular expression adjustment. Finally, a syntax fix in libtest.shlib prevents cleanup failures.

The new Alpine Linux runner

Alpine has a few peculiarities. One of them is ksh not being available in the repositories, so it is built from source, which takes a few minutes. Logging into the VMs also requires manually disabling cloud-init services during setup. The system is switched from mdev to eudev, and /etc/zfs/zpool.cache is created manually.

The remaining changes are what would be expected for a new Linux runner.

Possible future directions

This section lists ideas that could be explored in the future, but are likely out of scope for this pull request.

  • An Alpine Edge runner to test against upcoming releases. A stub already exists.
  • LLVM-based builds. LLVM-based user space builds work fine, but the Alpine kernel is built using GCC, so kernel modules cannot be built with LLVM.
  • Full (kernel + user space) LLVM-based builds.

How has this been tested?

Several test runs using GitHub Actions and the ZFS test suite. Alpine currently reaches a 94.62% pass rate.

All other distributions pass the CI pipeline after these changes.


Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Performance enhancement (non-breaking change which improves efficiency)
  • Code cleanup (non-breaking change which makes code smaller or more readable)
  • Quality assurance (non-breaking change which makes the code more robust against bugs)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
  • Documentation (a change to man pages or other documentation)

Checklist

@tonyhutter
Copy link
Contributor

tonyhutter commented Dec 29, 2025

While a fully green CI would be desirable prior to enabling the runner, this pull request deliberately limits its scope to adding Alpine support without enabling it for new pull requests. As such, the primary risk of merging lies in a small set of compatibility changes that affect existing runners.

You could add a textbox to the workflow_dispatch to specify a specific OS to run. This would allow you to run with alpine3-23 on-demand. It would look something like this (totally untested!):

diff --git a/.github/workflows/zfs-qemu.yml b/.github/workflows/zfs-qemu.yml
index db4fe3bdd..21ffe9aae 100644
--- a/.github/workflows/zfs-qemu.yml
+++ b/.github/workflows/zfs-qemu.yml
@@ -10,6 +10,12 @@ on:
         required: false
         default: ""
         description: "(optional) Experimental kernel version to install on Fedora (like '6.14' or '6.13.3-0.rc3')"
+      specific_os:
+        type: string
+        required: false
+        default: ""
+        description: "(optional) Only run on this specific OS (like 'fedora42' or 'alpine3-23')"
+
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
@@ -63,6 +69,9 @@ jobs:
               os_json=$(echo ${os_selection} | jq -c)
           fi
 
+          if {{ github.event.inputs.specific_os != '' }}; then
+              os_json='["'{{ github.event.inputs.specific_os }}'"]'
+          fi
           echo "os=$os_json" | tee -a $GITHUB_OUTPUT
           echo "ci_type=$ci_type" | tee -a $GITHUB_OUTPUT

alex-moch added a commit to alex-moch/zfs that referenced this pull request Dec 29, 2025
Pull request: openzfs#18087

Signed-off-by: Alexander Moch <mail@alexmoch.com>
Add an Alpine Linux 3.23 runner to the CI chain to run OpenZFS builds
and tests against musl libc.

Currently, zfs_send_sparse is killed after 10 minutes on Alpine, causing
cascading EBUSY failures in the test suite. With zfs_send_sparse
disabled, the ZFS test suite reaches a pass rate of 94.62%.

This commit introduces the required Alpine-specific setup and a small
set of shell and cloud-init compatibility fixes that also apply to
existing Linux runners.

The Alpine runner is not enabled by default and is not executed for new
pull requests.

Sponsored-by: ERNW Research GmbH - https://ernw-research.de/
Signed-off-by: Alexander Moch <amoch@ernw.de>
@alex-moch
Copy link
Contributor Author

Thanks for the input and merging #18085 so quickly; with that merged, the modules now build successfully on the Alpine runner.

I’ve rebased this pull request onto the current master branch and implemented the suggested changes.

Copy link
Contributor

@behlendorf behlendorf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed comment explaining the required changes. That definitely helped head off some questions I would have otherwise had! This LGTM. Once we're confident this builder will regularly pass the test suite we can also consider enabling it by default. Better to catch any build issues in the original PRs for new changes.

@tonyhutter
Copy link
Contributor

Your specific_os textbox works for me. I'm able to launch an on-demand alpine runner:
image

Please squash your commits and then I'll pull it in

@alex-moch
Copy link
Contributor Author

This pull request only has a single commit - nothing to squash.

There's probably some confusion because I have a separate development branch (alpine-ci-dev) with unsquashed commits in my fork. I referenced this PR in one of the commit messages, so it shows up in the history. I kept the PR branch (alpine-ci) squashed.

Copy link
Contributor

@mcmilk mcmilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Would be nice, if it can be added to the default runner list later.

@alex-moch
Copy link
Contributor Author

alex-moch commented Dec 30, 2025

Would be nice, if it can be added to the default runner list later.

I thought that getting this into master now and having several long-term maintainers take a look at it increases the chances it will soon make the remaining tests pass. 😉

@tonyhutter tonyhutter merged commit b9b8444 into openzfs:master Dec 30, 2025
26 checks passed
@alex-moch alex-moch deleted the alpine-ci branch December 30, 2025 17:34
mcmilk pushed a commit to mcmilk/zfs that referenced this pull request Jan 31, 2026
Add an Alpine Linux 3.23 runner to the CI chain to run OpenZFS builds
and tests against musl libc.

Currently, zfs_send_sparse is killed after 10 minutes on Alpine, causing
cascading EBUSY failures in the test suite. With zfs_send_sparse
disabled, the ZFS test suite reaches a pass rate of 94.62%.

This commit introduces the required Alpine-specific setup and a small
set of shell and cloud-init compatibility fixes that also apply to
existing Linux runners.

The Alpine runner is not enabled by default and is not executed for new
pull requests.

Sponsored-by: ERNW Research GmbH - https://ernw-research.de/

Signed-off-by: Alexander Moch <amoch@ernw.de>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Tino Reichardt <milky-zfs@mcmilk.de>
lundman pushed a commit to openzfsonosx/openzfs-fork that referenced this pull request Feb 5, 2026
Add an Alpine Linux 3.23 runner to the CI chain to run OpenZFS builds
and tests against musl libc.

Currently, zfs_send_sparse is killed after 10 minutes on Alpine, causing
cascading EBUSY failures in the test suite. With zfs_send_sparse
disabled, the ZFS test suite reaches a pass rate of 94.62%.

This commit introduces the required Alpine-specific setup and a small
set of shell and cloud-init compatibility fixes that also apply to
existing Linux runners.

The Alpine runner is not enabled by default and is not executed for new
pull requests.

Sponsored-by: ERNW Research GmbH - https://ernw-research.de/

Signed-off-by: Alexander Moch <amoch@ernw.de>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Tino Reichardt <milky-zfs@mcmilk.de>
tonyhutter pushed a commit to tonyhutter/zfs that referenced this pull request Feb 11, 2026
Add an Alpine Linux 3.23 runner to the CI chain to run OpenZFS builds
and tests against musl libc.

Currently, zfs_send_sparse is killed after 10 minutes on Alpine, causing
cascading EBUSY failures in the test suite. With zfs_send_sparse
disabled, the ZFS test suite reaches a pass rate of 94.62%.

This commit introduces the required Alpine-specific setup and a small
set of shell and cloud-init compatibility fixes that also apply to
existing Linux runners.

The Alpine runner is not enabled by default and is not executed for new
pull requests.

Sponsored-by: ERNW Research GmbH - https://ernw-research.de/

Signed-off-by: Alexander Moch <amoch@ernw.de>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Tino Reichardt <milky-zfs@mcmilk.de>
tonyhutter pushed a commit to tonyhutter/zfs that referenced this pull request Feb 11, 2026
Add an Alpine Linux 3.23 runner to the CI chain to run OpenZFS builds
and tests against musl libc.

Currently, zfs_send_sparse is killed after 10 minutes on Alpine, causing
cascading EBUSY failures in the test suite. With zfs_send_sparse
disabled, the ZFS test suite reaches a pass rate of 94.62%.

This commit introduces the required Alpine-specific setup and a small
set of shell and cloud-init compatibility fixes that also apply to
existing Linux runners.

The Alpine runner is not enabled by default and is not executed for new
pull requests.

Sponsored-by: ERNW Research GmbH - https://ernw-research.de/

Signed-off-by: Alexander Moch <amoch@ernw.de>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Tino Reichardt <milky-zfs@mcmilk.de>
lundman pushed a commit to openzfsonwindows/openzfs that referenced this pull request Feb 23, 2026
Add an Alpine Linux 3.23 runner to the CI chain to run OpenZFS builds
and tests against musl libc.

Currently, zfs_send_sparse is killed after 10 minutes on Alpine, causing
cascading EBUSY failures in the test suite. With zfs_send_sparse
disabled, the ZFS test suite reaches a pass rate of 94.62%.

This commit introduces the required Alpine-specific setup and a small
set of shell and cloud-init compatibility fixes that also apply to
existing Linux runners.

The Alpine runner is not enabled by default and is not executed for new
pull requests.

Sponsored-by: ERNW Research GmbH - https://ernw-research.de/

Signed-off-by: Alexander Moch <amoch@ernw.de>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Tino Reichardt <milky-zfs@mcmilk.de>
lundman pushed a commit to openzfsonwindows/openzfs that referenced this pull request Feb 23, 2026
Add an Alpine Linux 3.23 runner to the CI chain to run OpenZFS builds
and tests against musl libc.

Currently, zfs_send_sparse is killed after 10 minutes on Alpine, causing
cascading EBUSY failures in the test suite. With zfs_send_sparse
disabled, the ZFS test suite reaches a pass rate of 94.62%.

This commit introduces the required Alpine-specific setup and a small
set of shell and cloud-init compatibility fixes that also apply to
existing Linux runners.

The Alpine runner is not enabled by default and is not executed for new
pull requests.

Sponsored-by: ERNW Research GmbH - https://ernw-research.de/

Signed-off-by: Alexander Moch <amoch@ernw.de>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Tino Reichardt <milky-zfs@mcmilk.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants